The present invention relates to an image synthesizing apparatus for synthesizing a plurality of images, recorded by an electronic camera, having overlapping portions, on a computer, an image sensing apparatus used in combination with the image synthesizing apparatus, and an image synthesizing system using these apparatuses. More specifically, the present invention relates to automatic determination of timing for recording an image or images taken in series (image or images to be combined with an image or images which have already been recorded).
Processing for obtaining a wide angle image by synthesizing a plurality of images having overlapping portions using a computer is called "panoramic image synthesizing processing". This processing is to obtain a single image of stretched scenery or a single wide image.
The resolution of an electronic camera is lower than that of a silver halide camera or an image scanner (i.e., the number of pixels of the electronic camera is small), which is a defect of an electronic camera. Because of this defect, it is necessary to synthesize images recorded by an electronic camera into a panoramic image in order to obtain a wide image as well as an image of high resolution. An image of wider angle of view obtained in one shot with the electronic camera would provide lower the resolution than an image having normal width. In contrast, by sensing a single object (original image) in a plurality of shots, or sensing a scenery in a plurality of shots by shifting image sensing positions in the horizontal direction, the recorded image as a whole has as high resolution than that of an image sensed by a scanner or a silver halide camera.
Meanwhile, some panoramic image synthesizing systems have been configured with electronic cameras and personal computers connected to each other, which inputs images from the electronic cameras to the personal computers, and provided at affordable prices. In such systems, it is difficult for the computers to take full-size images on real-time basis because of the limitation of transmission paths, such as a parallel interface (I/F) or a serial I/F, connecting the electronic cameras and the computers. Therefore, in the conventional systems, images are reduced in size in order to secure a speed for transmitting images on almost a real-time basis and inputted by the computers in a short interval, then the images are provided for a user to confirm an image. Then, after the user confirms a scene which is desired to obtain a reduced image, a shutter button of the electronic camera is pressed. Accordingly, it is possible to obtain a full-size still image after confirming it. Note, the reduced image used for confirmation is called "confirmation image", hereinafter.
The most difficult process in generating a panoramic image by synthesizing a plurality of images is to find overlapping portions, i.e., to find a group of corresponding pixels in the plurality of images (the process is called "corresponding point extraction", hereinafter). Where corresponding points are to be found in two images, for example, position relationship between these two images is detected, thus it is possible to synthesize the images.
Upon performing corresponding point extraction, characterizing points and/or resemblance points are needed in these images. In a case where there are little characterizing points and/or resemblance points in the images, an error may occur when performing the corresponding point extraction. As a result, intervention by the user is required in many cases, and it is difficult to realize a full-automatic corresponding point extraction process with the conventional technique.